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Abstract 

A  method  for  reconstructing  3D  rational  B -spline  surfaces 
from  multiple  views  is  proposed.  The  method  takes  advan¬ 
tage  of  the  projective  invariance  properties  of  rational  B- 
splines.  Given  feature  correspondences  in  multiple  views, 
the  3D  surface  is  reconstructed  via  a  four  step  framework. 
First,  corresponding  features  in  each  view  are  given  an  ini¬ 
tial  surface  parameter  value  (s,t),  and  a  2D  B-spline  is 
fitted  in  each  view.  After  this  initialization,  an  iterative  min¬ 
imization  procedure  alternates  between  updating  the  2D  B- 
spline  control  points  and  re-estimating  each  feature’s  (s,  t). 
Next,  a  non-linear  minimization  method  is  used  to  upgrade 
the  2D  B-splines  to  2D  rational  B-splines,  and  obtain  a  bet¬ 
ter  fit.  Finally,  a  factorization  method  is  used  to  reconstruct 
the  3D  B-spline  surface  given  2D  B-splines  in  each  view. 
This  surface  recovery  method  can  be  applied  in  both  the 
perspective  and  orthographic  case.  The  orthographic  case 
allows  the  use  of  additional  constraints  in  the  recovery.  Ex¬ 
periments  with  real  and  synthetic  imagery  demonstrate  the 
efficacy  of  the  approach  for  the  orthographic  case. 

1  Introduction 

Recovering  models  from  multiple  views  is  an  area  of  great 
interest  in  computer  vision.  Algorithms  for  recovering  3D 
structure  and  camera  motion  have  many  practical  applica¬ 
tions,  such  as  computer  aided  design,  virtual  reality,  movie 
special  effects,  video  coding,  etc.  Many  previous  structure 
from  motion  algorithms  are  general  in  the  sense  that  they 
assume  no  prior  knowledge  about  the  scene.  However,  in 
practice,  the  scene  typically  contains  structures  with  strong 
geometric  regularities  that  can  be  used  to  constrain  the  esti¬ 
mation  problem.  Consequently,  algorithms  have  been  pro¬ 
posed  for  the  special  cases  of  planes  [1,  11,  22,  26,  27,  31], 
piecewise  planar  models  [3,  19],  polygonal  meshes  [14], 
and  quadric  surfaces  [8,  18]. 

In  this  paper  we  propose  a  method  for  reconstructing  3D 
rational  B-spline  surfaces  from  multiple  views.  As  will  be 
shown,  we  can  exploit  the  the  projective  invariance  proper¬ 
ties  of  rational  B-splines  to  gain  a  solution  to  the  3D  surface 
estimation  problem.  The  approach  is  demonstrated  in  a  for¬ 
mulation  for  recovering  a  quadratic  B-spline  surface  from 
point  correspondences  given  in  multiple  views.  The  for- 
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mulation  can  be  applied  in  both  the  perspective  and  ortho¬ 
graphic  case.  The  orthographic  case  allows  the  use  of  addi¬ 
tional  constraints  in  the  recovery.  Experiments  with  real  and 
synthetic  imagery  demonstrate  the  efficacy  of  the  approach 
for  the  orthographic  case. 

2  Related  Work 

Structure  recovery  algorithms  can  be  grouped  according  to 
the  complexity  of  their  shape  prior.  Algorithms  using  no 
shape  priors,  recover  3D  point  locations  directly.  One  ex¬ 
ample  of  this  is  due  to  [28],  where  image  measurements  are 
assembled  into  a  large  matrix  which  can  then  be  factorized 
to  reveal  the  underlying  geometry.  Others  have  extend  this 
approach  to  the  perspective  case  [21,  29].  In  contrast  with 
these  batch  processing  approaches,  others  have  developed 
methods  that  employ  a  recursive  estimation  theory  (e.g.,  a 
Kalman  filter)  to  estimate  3D  point  structure  and  camera 
parameters  in  an  on-line  fashion  [2,  4,  6,  16,  20].  This  ap¬ 
proach  allows  for  observation  and  process  noise  models  to 
be  directly  incorporated  into  the  reconstruction  method. 

Higher-level  constraints  of  the  structure  have  been  in¬ 
corporated  into  reconstruction  methods  as  well.  In  particu¬ 
lar,  the  special  case  of  planar  structure  has  been  extensively 
researched,  and  there  are  methods  that  incorporate  the  re¬ 
sulting  constraints  [26].  In  particular  [31]  has  developed 
a  closed-form  solution  to  planar  reconstruction  from  two 
views  and  [1]  had  extended  the  recursive  framework  of  [2], 
to  incorporate  planar  information.  In  [19]  Sinclair  focuses 
on  grouping  planar  regions,  while  [3]  focuses  on  direct  re¬ 
construction.  Some  methods  segment  the  scene  and  deter¬ 
mine  planes  for  each  patch  [1 1,  22,  27]. 

More  general  surface  representations  in  the  form  of 
meshes  were  explored  by  [14].  In  these  approaches  the  sur¬ 
face  is  found  by  minimizing  an  objective  function  relating 
the  estimated  surface  with  projections  into  estimated  cam¬ 
eras.  Surfaces  have  also  been  modeled  as  oriented  particles 
or  tiny  planes  with  associated  texture  in  [13,  15,  25].  While 
very  general,  mesh  and  particle  representations  tend  to  over 
parameterize  smoothly  curved  surfaces. 

In  [8]  and  [18],  methods  have  been  proposed  that  can  be 
used  when  objects  are  well-approximated  by  quadratic  sur¬ 
faces.  In  [8],  the  quadratic  surface  is  estimated  by  relating 
the  silhouette  of  the  quadratic  surfaces  to  the  projected  im¬ 
age.  In  a  different  approach,  [18]  examines  the  induced  flow 
field  of  quadratic  objects. 

In  this  work  we  further  develop  the  representation  of 
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smooth  surfaces  by  using  rational  B-splines  to  represent  3D 
surfaces  and  their  projections.  Spline  curve  representations 
have  been  used  extensively  in  computer  vision  in  the  context 
of  contour  tracking  and  contour  pose  recovery  (e.g.,  [7,  5]). 
Splines  have  also  been  used  in  motion  estimation  [24]  and 
image  registration  [23] .  Our  work  is  related  to  [23]  where 
a  depth  field  is  computed  from  spline-based  image  registra¬ 
tion.  In  that  approach,  the  spline  was  used  to  find  disparity 
maps  from  which  dense  depth  maps  were  then  computed.  In 
this  work,  we  instead  focus  on  directly  modeling  surfaces  in 
the  scene  as  3D  rational  B-splines. 


3  Rational  B-Splines 


Using  similar  notation,  rational  B-splines  take  the  form: 


R(s,  t) 


TV,  bk^k^k 

Sjfe=i  bkrWk 


(5) 


The  weights,  Wk  are  weight  factors  for  the  control  points. 
The  greater  the  value  of  Wk ,  the  closer  the  surface  is 
“pulled”  towards  the  control  point  P  k .  One  advantage  of 
the  rational  B- spline  representation  is  that  it  can  be  used  to 
represent  quadrics  exactly.  Another  advantage  is  that  they 
are  invariant  to  perspective  viewing  transformations,  as  will 
be  shown  in  Sec.3.1. 

Rational  B-splines  can  be  written  in  terms  of  a  homoge¬ 
neous  equation,  with  homogeneous  control  points: 


In  this  section  we  introduce  relevant  concepts  in  B-splines 
and  define  the  notation  used  in  this  paper.  For  a  detailed 
review  of  B-splines,  readers  are  directed  to  [10]. 

A  B -spline  surface  can  be  defined  in  terms  of  a  (ra  + 1)  x 
(n+ 1)  grid  of  control  points ,  and  a  set  of  blending  functions 
(interpolation  functions).  Given  these,  we  define  a  point  on 
the  B-spline  surface  at  a  particular  parameter  value  (s,t): 

m  n 

P(s,£)  =  Y',  Y'.  (1) 

i= 0  3=0 


P  *  = 


WkPk 

Wk 


(6) 


In  the  homogeneous  coordinate  system,  points  on  the  sur¬ 
face  are  represented: 


N 

R  (M)  =  JZ&fcPfc- 

k=l 


(7) 


3.1  Projection  of  Rational  B-Splines 


where  p ij  is  a  control  point,  and  Bi ^  and  Bj ^  are  blending 
functions  in  each  parametric  direction.  The  B-spline  blend¬ 
ing  functions  can  be  obtained  via  the  Cox  de  Boor  recursion 
formulas,  i.e.: 


Bj,  i  (s) 

Bj,d(s) 


1,  if  Sj  <  s  <  sj+i 

0,  otherwise 

S  —  Sj 


sj+d- 1  sj 
sj+d  ~  s 


+ 


sj+d  sj+ 1 


Bj,d— 1  is) 
Bj+l,d—l  (s)- 


(2) 


(3) 


The  polynomial  order  of  the  blending  functions  is  defined 
by  d.  The  influence  of  the  blending  functions  is  controlled 
by  a  knot  vector,  [sj]  (, Sj  <  Sj+i),  which  defines  the  region 
of  influence  of  each  blending  function.  In  our  implementa¬ 
tion,  we  will  set  d  =  3,  ra  =  n  =  4,  and  employ  a  uniform 
knot  vector:  [0, 0, 0, 1,  2, 3, 3, 3]. 

To  make  our  problem  formulation  easier,  we  will  need 
to  deviate  from  the  standard  B-spline  notation  as  follows. 
Given  the  set  of  N  =  (ra  +  1)  x  (n  +  1)  control  points,  we 
will  rewrite  (1): 


N 

P (s,t)  =  bkPk, 

k= 1 


(4) 


where  P&  =  p ij  and  bk  =  Bi^(t)Bj^(s),  such  that  i  = 
1  +  [k/(n  +  1)J  and  j  =  1  +  k  mod  (n  +  1). 


An  important  benefit  in  using  rational  B-splines  to  represent 
surfaces  becomes  apparent  when  we  consider  the  projection 
of  a  3D  surface  point  onto  the  image  plane  via  a  3  x  4  pro¬ 
jection  matrix  V.  A  point  on  the  rational  B-spline  R(s,  t)  is 
related  to  the  projected  image  point  u(s,  t)  via: 


Here  ~  is 
follows: 


—  VR(s,  t)  (8) 

defined  to  be  equality  up  to  scale.  From  (8)  it 


U  (s,t) 

1 


N 


V  ^2 

k= 1 
N 

Y;  bkVPk 

k= 1 
N 


(9) 

(10) 

(11) 


k= 1 


From  (1 1)  we  see  that  to  project  a  rational  spline  we  sim¬ 
ply  project  the  3D  surface’s  (homogeneous)  control  points 
to  obtain  a  set  of  projected  (homogeneous)  control  points 
for  the  2D  surface  (i.e.,  P^).  The  projected  surface  points 
are  generated  using  the  same  blending  coefficients  (bk). 

This  relation  suggests  a  method  for  reconstructing  a  B- 
spline  surface  given  multiple  views.  In  particular,  to  recon¬ 
struct  the  surface  we  simply  need  to  recover  the  projected 
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control  points,  P^,  of  the  2D  rational  spline  in  each  view. 
Since  these  control  points  are  in  principal  no  different  than 
other  points  we  can  employ  traditional  structure  from  mo¬ 
tion  methods  (e.g.,  [21,  28,  30])  to  obtain  the  corresponding 
3D  B-spline  control  points  P&  and  the  camera  parameters  if 
desired. 


4  Surface  Recovery 

Given  feature  correspondences  in  multiple  views,  the  goal 
will  be  to  reconstruct  the  3D  B-spline  surface.  It  is  as¬ 
sumed  that  the  image  location  of  M  features  and  their  cor¬ 
respondence  in  each  view  are  given  as  input  to  our  sys¬ 
tem.  For  a  particular  view,  the  location  of  the  ith  feature 
will  be  denoted  by  =  [ui,Uf]T.  To  reduce  the  com¬ 
plexity  of  the  surface  reconstruction,  the  order  of  the  spline 
surface  and  the  knot  vector  are  assumed  given.  As  men¬ 
tioned  earlier,  in  our  implementation  we  will  set  the  order 
of  the  spline  to  be  d  =  3,  and  m  —  n  —  4,  and  we  will 
use  a  uniform  knot  vector.  The  number  of  control  points 
N  =  (ra  +  1)  x  (n  +  1)  =  25.  We  must  therefore  recover 
the  N  control  point  locations,  as  well  as  the  surface  point 
parameters  ( Si,ti )  for  each  of  M  feature  points. 

As  will  be  detailed  in  the  rest  of  this  section,  the  3D 
surface  will  be  reconstructed  via  a  four  step  framework. 
First,  corresponding  features  in  each  view  are  given  an  ini¬ 
tial  surface  parameter  value  ( Si,U )  ,  and  a  2D  B-spline 
is  fitted  in  each  view.  After  this  initialization,  an  itera¬ 
tive  minimization  procedure  alternates  between  updating 
the  2D  B-spline  control  points  and  re-estimating  each  fea¬ 
ture’s  (si,ti).  Next,  a  nonlinear  minimization  method  is 
used  to  upgrade  the  2D  B- splines  to  2D  rational  B- splines, 
and  obtain  a  better  fit.  Finally,  a  factorization  method  is 
used  to  reconstruct  the  3D  rational  B-spline  surface  given 
2D  B- splines  in  each  view. 

4.1  Initializing  U) 

Given  a  collection  of  features,  an  initial  surface  parameter 
(. Si,U )  must  be  assigned  for  each  feature.  The  ith  feature 
is  assigned  the  same  ( Si,U )  in  all  views.  The  estimate  of 
each  feature’s  surface  parameter  will  be  improved  via  an  it¬ 
erative  procedure  later.  An  initial  ( Si,U )  parameter  value 
is  assigned  for  each  feature  in  the  first  view.  Correspond¬ 
ing  features  in  the  remaining  views  inherit  the  parameter 
assigned  in  the  first  view. 

The  surface  point  parameters  in  the  first  view  are  found 
by  placing  a  rectangular  grid  of  regularly  spaced  2D  control 
points  with  Wk  =  1  over  the  feature  points  in  the  first  image. 
The  overall  size  of  the  grid  is  determined  by  finding  the 
minimum  bounding  box  for  the  features.  This  results  in  a 
set  of  control  points  of  the  form  =  [ft*.,  1]T,  where 

(uk,Vk)  is  the  image  location  for  the  kth  control  point. 


The  surface  point  parameters  ( Si,U )  can  then  be  found 
by  searching  over  the  B-spline  parameter  space.  In  our  im¬ 
plementation,  this  is  done  by  sampling  uniformly  in  s  —  t 
space  and  choosing  the  sample  R(s/ ,  £/)  that  corresponds  to 
the  point  closest  to  the  feature  point,  u^: 

s*,t*  =  argmin  ||ui  -  R(s/,t/)||  (12) 

Sl,tl 

Sampling  is  then  performed  at  a  higher  resolution  around 
the  chosen  estimate,  to  find  an  improved  estimate 

that  is  closer  to  the  feature  via  (12).  This  is  repeated  until 
the  distance  between  the  B-spline  sample  and  the  feature  is 
within  a  specified  threshold. 


4.2  Refining  Control  Points  of  2D  B-Spline 


Given  an  initial  estimate  of  the  features’  parameter  values 
(si  ,ti),  the  control  points  in  each  of  the  views  are  estimated. 
In  this  step,  the  control  points  are  recovered  independently 
for  each  view,  and  the  third  component  of  the  control  points 
is  fixed  Wk  =  1.  For  a  given  (si,ti),  the  point  on  the  B- 
spline  can  be  computed: 


R{si,ti)T 


1 - '  > 

-o 

-o 

1 - 'J 

i 

" 

_ 1 

V 

bf 

1 

.  b* 

•On 

(13) 


where  is  the  vector  of  blending  coefficients  for  ( Si,U ). 

Ideally  we  would  like  the  feature  point  observations  to 
match  their  assigned  R(s*,ti).  If  this  were  the  case  the 
observations  would  be  related  to  the  control  points  via: 


Ul  Vi 

bf  1 

Ul  Vl 

u2  V2 

— 

bf 

u2  v2 

Um  VM 

1 

_ i 

UN  Vn 

(14) 


This  can  be  rewritten  in  the  standard  matrix  vector  form  if 
we  denote: 


Ul 

Ul 

U2 

U2 

um 

UN 

5  X  = 

Ul 

Ul 

u2 

U2 

vm 

UN 

Then  (14)  becomes 

y  =  Bx 


where: 


B  = 


B  0 
0  B 


(15) 


(16) 

(17) 
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Since  (16)  will  not  be  satisfied  exactly  due  to  measurement 
noise  and  error  in  assigning  (si,ti),  we  seek  the  solution 
that  minimizes  the  mean  squared  error. 

x*  =  argmin  ||y  —  Bx||2  (18) 

X 

which  can  be  found  using  conjugate  gradient  descent  [17]. 

4.3  Updating  ( tj) 

With  updated  control  points  we  can  proceed  to  improve 
each  feature’s  assigned  surface  parameter,  ( Si,U ).  Recall 
that  corresponding  points  have  the  same  parameters  in  all 
views.  In  refining  our  estimate  of  a  feature’s  we 

seek  to  minimize  the  sum  of  the  distance  between  the  hy¬ 
pothesized  point  and  the  observed  feature  location  over  all 
views: 

Nv 

s*,t*  =  argmin^  ||u£°  -  R«(si,ii)||2  (19) 

3  =  1 

where  Nv  is  the  number  of  views,  is  the  observed  fea¬ 
ture  location  in  the  jth  view,  and  )  is  the  spline 

predicted  point  location  for  that  view.  This  can  be  solved  ef¬ 
ficiently  using  the  previous  estimate  (s«,  U)  as  starting  point 
in  a  minimization  algorithm  that  is  based  on  golden  section 
search  and  parabolic  interpolation  [12]. 

Given  improved  estimates  of  ( Si,U )  obtained  through 
the  nonlinear  minimization,  we  can  then  refine  the  estimates 
of  the  control  points  in  each  view  as  described  in  Sec.  4.2. 
As  suggested  in  [9],  these  two  steps  can  be  applied  alter¬ 
nately,  first  updating  the  B -spline  control  points  and  then 
re-estimating  each  feature’s  ( Si,U ).  In  our  experience,  we 
have  observed  that  three  or  four  iterations  are  needed  for 
reasonable  convergence. 

4.4  Upgrade  to  Rational  B -Splines 

Recall  that  up  until  this  point,  the  control  point  weights  have 
all  been  set  Wk  =  1.  Estimates  of  the  the  2D  spline  con¬ 
trol  points  have  been  obtained  in  each  view,  keeping  the 
weights  fixed  as  described  above.  We  can  now  use  a  nonlin¬ 
ear  minimization  technique  to  further  fit  the  spline  in  partic¬ 
ular  view,  by  allowing  the  weights  to  vary.  For  a  particular 
view,  each  control  point’s  weight  is  chosen  to  satisfy: 

M 

H  =  argmin  V  ||uj  -  R(s;,i;)||2  (20) 

p * 

In  practice,  we  can  keep  the  point  parameters  while  finding 
the  weights.  The  solution  is  found  using  the  Levenberg- 
Marquardt  [17]  nonlinear  minimization  procedure.  The  so¬ 
lution  is  computed  separately  for  each  view. 


4.4.1  Special  Case:  Orthography 

When  the  cameras  are  orthographic  the  project  matrix  takes 
the  form: 

abed 

V=  e  f  g  h  (21) 

_  °  °  °  1  _ 

In  this  case,  the  Wk  component  of  the  control  points  of 
the  2D  B -splines  across  all  views  are  constrained  to  be  the 
same.  This  additional  constraint  can  be  used  when  fitting 
the  2D  rational  B- splines.  In  this  case,  the  objective  func¬ 
tion  to  be  minimized  should  include  the  control  points  in 
all  views  simultaneously,  since  the  kth  control  point  in  all 
views  share  the  same  Wk. 

In  addition,  if  the  cameras  are  orthographic  and  it 
is  known  that  the  observed  3D  surface  can  be  well- 
approximated  by  a  non-rational  B-spline,  then  the  “upgrade 
to  rational  B- splines”  step  can  be  omitted  since  the  weights 
are  known  to  all  be  Wk  =  1. 

4.5  Reconstruction 

Given  recovered  2D  control  points  for  rational  B- splines  in 
all  views,  we  can  employ  standard  factorization  algorithms 
[28,  21]  to  recover  the  3D  shape  and  the  camera  matrices. 
The  recovered  2D  control  points  are  assembled  into  a  mea¬ 
surement  matrix.  This  matrix  is  then  factorized  using  SVD 
to  reveal  its  projective  structure.  Following  this,  a  homog- 
raphy  is  applied  to  both  the  cameras  and  the  structure  to 
transform  the  projective  reconstruction  to  an  Euclidean  one. 

4.5.1  Perspective  Reconstruction 

In  the  perspective  case  we  assemble  the  homogenized  con¬ 
trol  points  into  a  measurement  matrix  that  relates  to  the  3D 
B-spline  surface  control  points  and  camera  matrices  as: 

-p^i)  -p^i)  r  'p(i) 
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Typically  in  perspective  reconstructions  of  this  form  the  ho¬ 
mogenous  component  of  each  point  is  not  known  and  must 
also  be  recovered.  However,  in  our  case  the  s  have  al¬ 
ready  been  found  and  can  be  incorporated  directly.  We  can 
then  assemble  the  recovered  control  points  P^  into  the 
measurement  matrix  on  the  left-hand  side  of  (22).  The  cam¬ 
era  matrices  and  3D  control  points  are  then  recovered  as 
described  in  [21]. 

4.5.2  Orthographic  Reconstruction 

Reconstruction  in  the  orthographic  case  is  similar  to  that  of 
the  perspective  case.  However,  here  we  are  interested  in 


the  2  x  3  representation  of  the  camera  matrices  M .  To  use 
this  representation  we  need  to  substract  off  the  centroid  of 
the  feature  points  in  each  view,  which  effectively  sets  the 
principal  point  to  zero.  Thus  M  is  the  upper  left  2x3 
submatrix  of  V. 

In  this  framework  the  reconstruction  is  found  by  relating 
the  observations  to  the  cameras  and  3D  control  point  loca¬ 
tions  through: 


(a)Sample  views.  Feature  observations  are  marked  as  crosses  and 
2D  spline  predicted  locations  marked  as  circles. 
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where  are  the  recovered  [u^\v^]  components  of 

each  control  point  P .  The  camera  matrices  and  3D  con¬ 
trol  points  are  then  recovered  as  described  in  [28].  The 
weights  Wk  of  the  control  points  are  already  known,  as  they 
were  obtained  in  fitting  the  2D  rational  splines  (Sec.  4.4.1). 


Figure  1 :  Synthetic  Reconstruction  results  with  seven  Views,  100 
uniformly  sampled  feature  points  and  noise  with  a  =  .4107 


4.6  Minimum  Number  of  Features  and  Views 

In  fitting  the  2D  rational  B- Splines,  there  are  a  total  of  N 
control  points  with  three  degrees  of  freedom  for  each  of  Nv 
views.  In  addition,  there  are  two  surface  parameters  {si,ti) 
for  each  of  M  observations.  Each  feature  observation  pro¬ 
vides  two  constraints.  Therefore,  in  general  we  need: 

3xjVx^  +  2xM<2xMxiVi;.  (24) 

In  our  implementation,  biquadratic  surfaces  were  used; 
therefore,  in  our  case  N  =  25. 

For  the  final  reconstructed  surface  we  have  N  3D  control 
points  as  well  as  the  (s^,  ti)  for  each  of  M  observed  points. 
Therefore,  in  the  final  reconstruction  step  we  need: 

4xN  +  2xM<2xMxNv.  (25) 

For  reconstruction  we  need  the  numbers  of  views  and 
observed  feature  points  to  satisfy  both  (24)  and  (25). 

5  Experiments 

To  evaluate  our  system,  reconstructions  were  performed  on 
real  and  synthetic  data  sequences.  These  sequences  repre¬ 
sent  a  variety  of  different  shapes  that  can  be  represented 
using  rational  biquadratic  spline  patches. 

5.1  Synthetic  Sequences 

The  synthetic  experiments  were  performed  by  generating 
observations  of  a  biquadratic  rational  B -Spline  object  with 
control  points  drawn  from  a  parabolic  function.  A  set  of 
100  surface  features  were  generated,  sampled  uniformly  in 


Figure  2:  Average  reconstruction  error  for  us.  noise  level. 

(s,  t).  The  orthographic  projections  of  the  feature  set  were 
generated  from  seven  projection  matrices  that  corresponded 
to  different  cameras  viewpoints.  Gaussian  noise  was  then 
added  to  the  2D  feature  locations. 

Examples  of  the  generated  2D  views  are  shown  in  Fig. 
1(a).  From  these  views  an  estimate  of  the  surface  was  ex¬ 
tracted  using  our  method.  The  estimate  was  then  aligned 
with  the  true  object  by  finding  the  rigid  transform  and 
isotropic  scaling  that  minimized  the  distance  between  corre¬ 
sponding  3D  points.  This  was  necessary  as  the  reconstruc¬ 
tion  is  only  defined  up  to  a  scale  factor  and  a  rigid  transform. 
Example  reconstruction  results  are  shown  in  Fig.  1. 

The  sensitivity  of  the  reconstruction  method  was  tested 
by  varying  the  standard  deviation  of  the  added  noise.  A 
graph  of  results  of  this  experiment  is  shown  in  Fig.  2.  Here 
we  show  the  average  reconstruction  error  for  a  given  level 
of  noise,  where  the  error  is  the  average  distance  between 
corresponding  points  of  the  aligned  surface  estimate  and  the 
true  surface  computed  from  multiple  instances  of  the  noise 
level.  From  this  plot  we  see  that  our  approach  is  reasonably 
stable  with  respect  to  additive  noise. 


5 


(a)Sample  views  with  feature  observations  as  crosses  and  2D  rational- spline  predicted  locations  as  circles. 


(b)  Reconstructed  patch  (wireframe) 


(c)  Actual  object 


(d)  Reconstructed  patch  (shaded) 
from  similar  view 


Figure  3:  Reconstruction  of  bottle  sequence  from  101  features  in  four  views. 


5.2  Real  Sequences 

A  number  of  real  sequences  were  taken  of  smooth  textured 
objects  using  a  USB  camera  with  resolution  of  640  x  480. 
In  these  sequences  orthographic  cameras  were  assumed  and 
only  a  patch  of  the  object  was  considered  for  reconstruction. 

Corresponding  features  were  extracted  by  selecting 
points  in  one  frame  and  tracking  them  with  an  iterative  pyra¬ 
midal  Lucas-Kanade  tracker  that  is  available  in  OpenCV 1 . 
Following  this,  a  few  frames  with  disparate  views  were  se¬ 
lected  for  use  in  reconstruction.  Features  in  each  selected 
input  frame  are  shown  in  Figs.  3,  4,  and  5. 

In  the  first  sequence  shown  in  Fig.  3,  four  views  with  101 
features  of  the  object  were  taken.  Samples  of  the  fitted  2D 
rational- Spline  predicted  feature  locations  are  shown  in  Fig. 
3(a)  along  with  the  observed  feature  points.  Note  that  the 
cylindrical  cross-section  of  the  bottle  is  actually  a  rounded 
triangular  shape.  From  this  figure  we  see  that  we  were  able 
to  accurately  fit  the  2D  rational  B-splines.  The  subsequent 
3D  surface  reconstruction  for  this  patch  of  the  object,  along 
with  the  an  image  of  the  actual  object  are  shown  from  a 
similar  viewpoint  are  shown  in  3(b,c).  From  these  plots  we 
see  the  reconstructed  surface  appears  similar  to  the  true  ob¬ 
ject,  though  there  are  some  artifacts  around  the  edges  of  the 
spline.  This  resulted  from  trimming  the  parts  of  the  spline 
surface  that  were  outside  the  convex  set  of  observed  points. 

Additional  example  reconstructions  are  shown  in  Figs.  4 
and  5.  In  Fig.  4  the  surface  of  a  mushroom  cap  was  recon¬ 
structed  from  four  views  and  105  features.  As  seen  in  Fig. 
4(b,c),  the  resulting  3D  surface  reconstruction  accurately 

1http://www.intel.com/research/mrl/research/opencv/ 


includes  a  slight  dimple  in  the  middle  of  the  mushroom  cap 
surface.  An  example  of  reconstruction  for  a  concave  sur¬ 
face  is  shown  in  Fig.  5.  In  this  case,  seven  views  and  127 
features  were  used  as  input  to  the  reconstruction. 

6  Discussion  and  Future  Work 

As  exhibited  in  the  experiments  this  approach  is  able  to  ex¬ 
tract  the  shape  of  surfaces  modeled  as  3D  rational  spline 
patches.  While  the  basic  shape  of  the  object  was  recov¬ 
ered,  the  visual  quality  of  the  reconstructions  depends  on 
how  well  features  were  extracted.  In  particular  solutions 
exhibit  oscillatory  in  places  of  the  surface  when  there  were 
not  many  or  poorly  tracked  features.  This  behavior  is  ex¬ 
pected,  and  to  compensate,  penalties  on  smoothness  need  to 
be  incorporated.  This  can  be  done  by  adding  smoothness 
terms  in  the  reconstruction  step  [9].  Despite  this,  results  of 
this  method  are  promising. 

In  future  work  we  hope  to  enhance  this  system  in  several 
ways.  First,  we  plan  to  incorporate  features  estimation  into 
the  framework.  Currently,  feature  estimation  is  a  separate 
step.  This  method  however,  requires  the  use  of  many  fea¬ 
tures  and  finding  them  is  often  difficult.  We  expect  better 
overall  performance  if  feature  tracking  is  incorporated  di¬ 
rectly  within  surface  estimation.  By  doing  this  we  may  bet¬ 
ter  cope  with  occlusions  and  incorporate  other  image  fea¬ 
tures,  such  as  lines  and  the  silhouette  edges  of  the  surface. 
We  also  plan  to  extend  the  surface  representation  to  deal 
with  more  complex  surfaces,  by  extending  the  formulation 
from  a  single  spline  patch  to  piecewise  spline  surfaces.  Ad¬ 
ditionally,  surface  creases  may  also  be  considered. 
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(a)Sample  views  with  feature  observations  as  crosses  and  2D  rational- spline  predicted  locations  as  circles. 


(b)  Reconstructed  patch  (wireframe)  (c)  Actual  object  (d)  Reconstructed  patch  (shaded) 

from  similar  view 

Figure  4:  Reconstruction  of  mushroom  cap  surface  from  105  features  in  four  views. 


(a)Sample  views  with  feature  observations  as  crosses  and  2D  rational- spline  predicted  locations  as  circles. 


(b)  Reconstructed  patch  (wireframe)  (c)  Actual  object  (d)  Reconstructed  patch  (shaded) 

from  similar  view 

Figure  5:  Reconstruction  of  concabe  3D  rational  B-spline  surface  from  seven  views  and  127  features. 
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